1. Field of The Invention
The present invention relates to hardware implementations of real-time neural network systems.
2. Description of the Related Art
Neural network models have been developed in an effort to achieve improved performance in areas such as pattern recognition, classification and image processing. These neural networks, which follow the pattern of biological neural nets, include a plurality of neural layers each having a plurality of neurons. Each of the neurons within a neural layer outputs a response to a weighted sum of inputs. Specifically, a neuron will receive a plurality of inputs, amplify or decrease each input by a corresponding weight, and output a value in accordance with the transfer function of the neuron operating on the weighted sum of inputs. In contrast to microprocessor-based systems which are limited by their von Neumann and sequential architecture, a neural network can perform a variety of non-linear computations at relatively high speeds due to its relatively large degree of parallelism and connectivity.
Despite the appeal of neural networks and their highly-parallel architecture, few low-cost hardware implementations exist for the neural networks because of their inherent requirements, namely a high degree of connectivity and accurate, high-resolution neural response characterizations. One implementation of the neural network executes the neural network using a software-based system. Thus, since the neural network is emulated in software, the hardware implementation still relies upon a microprocessor-based system using a central processing unit having a von Neumann architecture. Thus, the software emulator of the neural network is subject to the limitations of the microprocessor which maintains a von Neumann architecture, thus resulting in a slower speed.
Further limitations in the implementation of neural networks are caused by the high cost of implementing such a highly-connective neural network in hardware. Any attempt to implement a neural network on a semiconductor chip greatly increases the area of the IC, thereby increasing not only the processing time but also the cost to manufacture the integrated circuit. Further, a neural network implemented in hardware becomes less flexible to the learning which is required before any pattern recognition by the neural network can be effective.
A final difficulty with conventional hardware-based neural networks is that if a digital implementation is preferred for greater efficiency, the flexibility in implementing accurate neural response characteristics is limited. For example, it may be desirable that a neuron has a sigmoidal response to the weighted sum of inputs. However, any desire to improve the accuracy of the sigmoidal response function necessitates a higher number of digital bits to represent the digital number, thereby increasing the complexity and the cost of the hardware-based neural network.
It would be desirable to provide an apparatus which provides maximum flexibility for constructing diverse neural networks. It would also be desirable to provide fast network processing times and to minimize the cost of the IC chip by reducing the area of the die.
Further, it would be desirable to provide an apparatus which implements a neural network while enabling flexible learning of the neural network as well as fast, efficient data transfer into the neural network.
Finally, it would be desirable to provide an apparatus which performs complex neuron response functions with high resolution and which is implemented economically.